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1 Abstract 

(N 
> 

Models of cortical neuronal circuits commonly depend on inhibitory feedback to control gain, pro- 
vide signal normalization, and to selectively amplify signals using winner-take-all (WTA) dynamics. 
Such models generally assume that excitatory and inhibitory neurons are able to interact easily, 
because their axons and dendrites are co-localized in the same small volume. However, quanti- 
tative neuroanatomical studies of the dimensions of axonal and dendritic trees of neurons in the 
neocortex show that this co-localization assumption is not valid. In this paper we describe a simple 
modification to the WTA circuit design that permits the effects of distributed inhibitory neurons 
to be coupled through synchronization, and so allows a single WTA to be distributed widely in cor- 
tical space, well beyond the arborization of any single inhibitory neuron, and even across different 
cortical areas. We prove by non-linear contraction analysis, and demonstrate by simulation that 
distributed WTA sub-systems combined by such inhibitory synchrony are inherently stable. We 
show analytically that synchronization is substantially faster than winner selection. This circuit 
mechanism allows networks of independent WTAs to fully or partially compete with other. 
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2 Introduction 

Many models of neuronal computation involve the interaction of a population of excitatory neu- 
rons whose outputs drive inhibitory neuron(s), which in turn provide global negative feedback to 

*This article has been accepted by MIT Press for publication in a future issue of Neural Computation (2012). 
This is a pre-print version (as accepted). This version was updated on 04/03/12, correcting minor typos discovered 
during proof-reading. 
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the excitatory pool (Amari & Arbib, 1977; Douglas, Koch, Mahowald, Martin, & Suarez, 1995; 
Hahnloser, Sarpeshkar, Mahowald, Douglas, & Seung, 2000; Yuille & Geiger, 2003; Maass, 2000; 
Hertz, Krogh, & Palmer, 1991; Rabinovich et al., 2000; Rutishauser, Douglas, & Slotine, 2011; 
Coultrip, Granger, & Lynch, 1992). Practical implementation of such circuits in biological neural 
circuits depend on co-localization of the excitatory and inhibitory neurons, an assumption which 
studies of the extents of axonal and dendritic trees of neurons in the neocortex show is not valid 
(Katzel, Zemelman, Buetfering, Wolfel, & Miesenbock, 2011; Binzegger, Douglas, & Martin, 2004; 
Shepherd, Stepanyants, Bureau, Chklovskii, & Svoboda, 2005; Douglas & Martin, 2004). Firstly, 
a substantial fraction of the axonal arborization of a typical excitatory 'spiny' pyramidal neuron 
extends well beyond the range of the arborization of a typical 'smooth' inhibitory neuron, partic- 
ularly in the populous superficial layers of the neocortex (Yabuta, 1998; Binzegger et al., 2004). 
This spatial arrangement means that excitatory effects can propagate well outside the range of 
the negative feedback provided by a single inhibitory neuron. Secondly, the horizontally disposed 
'basket' type of inhibitory neuron, which is a prime candidate for performing normalization, makes 
multiple synaptic contacts with its excitatory targets, so that even within the range of its axonal 
arborization, not all the members of an excitatory population can be covered by its effect. This 
connection pattern means that excitatory neurons within some local population must either be 
partitioned functionally, or multiple smooth cells must co-operate to cover the entire population 
of excitatory cells. 

In previous publications we have shown how winner-take- all (WTA) circuits composed of a small 
population of excitatory neurons and a single inhibitory neuron can be combined to construct super- 
circuits that exhibit finite state-machine (FSM) like behavior (Rutishauser & Douglas, 2009; Neftci, 
Chicca, Indiveri, Cook, & Douglas, 2010). The super-circuits made use of sparse excitatory cross- 
connections between WTA modules to express the required states of the FSM. These excitatory 
connections can extend well outside of the range of the local WTA connections, and so are consistent 
with the observed long-range lateral excitatory connections referred to above. On the other hand, 
we have not previously confronted the question of whether the WTA is necessarily localized to 
the extent of the smooth-cell arborization, or whether the WTA can itself be well distributed in 
space within or between cortical lamina, or even between cortical areas. In this paper we describe 
a simple modification to the WTA circuit design that couples the effects of distributed inhibitory 
neurons through synchronization, and so permits a WTA to be widely distributed in cortical 
space, well beyond the range of the axonal arborization of any single inhibitory neuron, and even 
across cortical areas. We also demonstrate that such a distributed WTA is inherently stable in its 
operation. 

3 Results 

We have considered a number of circuits that could be used to distribute spatially the WTA 
behavior (Fig 1). However, we will describe and analyze only the circuit shown in Fig. 2, which 
we consider to be the most elegant of the distributive mechanisms (notice the similarity to Fig. 
IB). The key insight is the following: Under normal operating conditions, all the participating 
distributed inhibitory neurons should receive the same summed excitatory input. We achieve this 
by interposing an excitatory neuron in the negative feedback loop from the excitatory population 
to its local inhibitory neuron. Instead of the local inhibitory neuron summing over its excitatory 
population, the interposed neuron performs the summing and passes its result to the inhibitory 
neuron. This result is also copied to the more distant inhibitory neurons in the spatially distributed 
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Figure 1: Circuits for distributing WTAs. (A) Illustration of the principal idea - mutual excitation 
of the inhibitory neurons. (B-E) Are are biologically plausible versions. (B) Implementation using 
intermediate excitatory neurons Z\$- This circuit will be considered in detail in Fig. 2B with more 
realistic connectivity. (C) Implementation using disinhibition of persistently active units z\$, as 
illustrated by the step-functions. (D) Implementation with disinhibition and long-range excitatory 
units. (E) Implementation using multiplication of inhibitory neurons. Here, 23 = 2/3 at all times. 
The maximal excitatory projection length is double that of the inhibitory. 
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Figure 2: Schematic of connectivity. Gray units and dashed lines are inhibitory, white units and 
straight lines are excitatory. (A) Single WTA with two possible winners X\^, inhibitory unit x$, 
and intermediate excitatory unit £4 that carries the average activity of x\^- (B) Coupling of two 
WTAs to form a single WTA with 4 possible winners. /?4 are excitatory long-range connections 
that serve to synchronize the two inhibitory units. (C-D) Are reduced versions for theoretical 
analysis. (C) Merged WTA with one winner each and thus two possible winners X\ and y\ in the 
merged network. (D) Reduced single WTA. 

WTA. In this way the inhibitory neuron of each sub- WTA sums over the projections from the 
interposed excitatory neurons of all other sub- WTAs, including its own one. Thus, each inhibitory 
neuron is able to provide feedback inhibition to its local sub- WTA that is proportional to the total 
excitation provided by all excitatory neurons participating in the entire distributed WTA. We will 
show that functionally this amounts to a form of synchrony between all the inhibitory units. 

3.1 Connectivity and dynamics - single WTA 

All the circuits of Fig 1 can achieve a distributed WTA by merging several independent WTAs, 
but we consider the circuit shown in Fig 2B to be most feasible, and so our analysis will focus 
on this one. However, similar reasoning could be applied to all the variants shown. Note that 
our chosen circuit is similar to that of Fig IB, but has a more realistic connectivity pattern in 
that the summed excitatory activity is only projected onto a single unit, which requires less wiring 
specificity than Fig IB. 

The dynamics of a single WTA (Fig 2A) with in total N units, consisting of 1..N — 2 excitatory 
units, one inhibitory unit xn_\ and one intermediary interconnect excitatory unit Xn, are 

rii + Gxi = f(Ii + axi - ftijv-i - Ti) 

TXjsr-l + Gxn^i = f(@3XN — T N _i) 

N-2 

tx n + Gx N = f((3 2 ^2 x o ~ W 

Each excitatory unit receives recurrent input from itself (a?i) and its neighbors {0:2,3,..., see Fig 
2A). For simplicity, only self-recurrence is considered here (a = «i and 0:2,3,... — 0), but very similar 
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arguments obtain when recurrence from neighboring units is included. Using the weight matrix 
W the dynamics of this system is described as 



rx + Gx = /(Wx - T + I(t)) 



(2) 



with 
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where the order of units is £1,2,3,4 (i-e. first column and row is Xi, last column/row is rr 4 ). The 
firing rate activation function f(x) is a non-saturating rectification non-linearity max(0, x). We 
assume r = 1 and G — 1 throughout unless mentioned otherwise. T = [T±, Tjv-i, Tjv] is a vector 
of the constant activation thresholds Tj > 0. 

3.2 Connectivity and dynamics - coupled WTA 

Two identical single WTAs, each described by weight matrices Wi = W2 = W can be combined 
into one distributed WTA that acts functionally as a single WTA by adding a recurrent excitatory 
feedback loop /3 4 between the two WTAs (Fig 2B). The weight matrix W c of the merged system 
is 

Wi C 2 
Ci w 2 



(4) 



with interconnections 



Ci 







/3 4 





(5) 



The dynamics of this system are as shown in Eq 2 using W = Wc. 



3.3 Stability analysis 

The stability analysis, using non-linear contraction analysis (see Appendix) (Lohmiller & Slotine, 
1998, 2000; Slotine, 2003; W. Wang & Slotine, 2005), consists of three steps: i) demonstrate con- 
traction of a single WTA, ii) merge two WTAs by demonstrating that inhibitory units synchronize, 
and iii) demonstrate contraction of the combined WTAs. We have previously shown how contrac- 
tion analysis can be applied to reasoning over the stability and functionality of WTA circuits 
(Rutishauser et al., 2011). Here, we apply and extend the same methods to this new circuit. 

Contraction analysis is based on the Jacobians of the system. For purposes of analysis, but 
without loss of generality, we will base this section on a reduced system with only 1 possible winner 
for each WTA as shown in Fig 2C. 

The Jacobian of a single system is 



rJ = 



l±a — 1 




Il-Pl 

-1 







I2P3 
-1 



(6) 
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In a stable network, a constant external input to the first unit x\ will lead to a constant amplitude 
of x\ that is either an amplified or suppressed version of its input. 

The activation function f(x) = max(x, 0) is not continuously differentiable, but it is continuous 
in both space and time, so that contraction results can still be applied directly (Lohmiller & Slotine, 
2000). Furthermore, the activation function is piecewise linear with a derivative of either or 1. 
We exploit this property by inserting dummy terms lj, which can either be or 1 according to the 
derivative of f(x): lj = 4-f(xj(t)). In this case, all units are active and thus h = l 2 = h = 1- 

A system with Jacobian J is contracting if 

J 1 < (7) 

where is a constant transformation into an appropriate metric and F = J 1 is the gen- 
eralized Jacobian. If F is negative definite, the system is said to be contracting. We have shown 
previous (Rutishauser et al., 2011) (section 2.4) how to choose the constant transformation and 
conditions that guarantee contraction for a WTA circuit where all excitatory units provide direct 
input to the inhibitory unit (Fig 1A). In summary, = Q 1 where Q is defined based on the 
eigendecomposition J = QAQ -1 . In this case 

< a < 1\/~{h{h 

< /5i/5 2 < 1 (8) 

guarantee contraction for any such WTA of any size (Rutishauser et al., 2011). 

Structurally, the two versions of the WTA are equivalent in that an additional unit was added 
in the pathway of recurrent inhibition, but no inhibition is added or removed (Compare Fig 1A to 
Fig 2A). Thus, we can apply the same constraints by replacing the product fi\f$i with in 
above equations. This product is equivalent to the inhibitory loop gain. This reduction is verified 
as follows. Using the notation shown in Fig 2C, assume that T = for x 3 where T > for the 
other units. Then, 

X 3 + X 3 = f(02Xl) 

x 2 + x 2 = f{(3 3 x 3 - T 2 ) (9) 

At steady-state, x 2 = /(/^/^i), showing that x 3 and x 2 can be merged into a single unit x 2 by 
providing input of weight P 2 p 3 directly to unit x 2 (Fig 2D). The first key result of this paper are 
the following limits for contraction of a single such WTA (Fig 2A): 

< a < 2 y /p 1 p 2 p 3 

< /Wa < 1 (10) 

3.3.1 Synchronizing two WTAs 

Next, we show that connecting two WTAs in the manner illustrated in Fig 2C results in synchro- 
nization of the two inhibitory units, which in turn leads to the two WTAs merging into a single 
WTA. Note that by synchronization, we mean that two variables have the same trajectory, or more 
generally that their difference is constant (in contrast to other meanings of synchronization i.e. in 
population coding). The approach is to show that adding excitatory connections ^4 of sufficient 
strength will lead to the activity of the two inhibitory units x 2 and y 2 approaching a constant 
difference. 
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The Jacobian of the coupled system as shown in Fig 2C is 



with Ji = J 2 = J (see Eq 6) and 



rD 1; 



Ji D 2 
Di J 2 





Z 2 ,5/3 4 




(11) 



(12) 



Following (Pham & Slotine, 2007; Rutishauser et al., 2011), synchronization occurs exponentially 
if the following holds: 



VJ C V < 



(13) 



where V defines an invariant subset of the system such that Vx is constant, with x = (xi, x 2 , x%, yi, y 2 , Vs)- 
Here, we define synchrony as a regime where the differences between the inhibitory units x 2 — y 2 
and between the interconnect units x 3 — y 3 are constant (although not necessarily zero). This 
results in 

"0100-1 
l 1 -1 

which embeds the two conditions. 
Condition (13) is satisfied if 



V = 



(14) 



1 < a 
< (3 4 < & + 2 
/3 3 <2 



(15) 
(16) 



The conditions on the interconnect-weight /?4 guarantees that the dynamics are stable and that the 
inhibitory units synchronize. As long as /3 4 > is sufficiently small but non-zero, the inhibitory 
parts of the system will synchronize. Realistically, (3^ needs to be sufficiently large to drive the other 
inhibitory neuron above threshold and will thus be a function of the threshold T (see (Rutishauser 
et al., 2011), Eq 2.51). Here, synchrony is defined as their difference being constant. This in 
turn shows that the two WTAs have merged into a single WTA since the definition of a WTA is 
that each excitatory unit receives an equivalent amount of inhibition (during convergence but not 
necessarily afterwards, see simulations). This is our second key result. 



3.3.2 Stability of pairwise combinations of WTAs 

The final step of the stability analysis are conditions for the coupling strength /?4 > such that 
the coupled system as shown in Fig 2C is contracting. The reasoning in this section assumes that 
the individual subsystems are contracting (as shown above). 

The Jacobian of the combined system remains Eq 11, where Ji 2 are the Jacobians of the indi- 
vidual systems and Ci )2 are the coupling terms. Rewriting the Jacobian of the second subsystem 
J 2 by variable permutation y' 2 = y 3 and y' 3 = y 2 allows expression of the system in the form of 



Jl 


D 2 ~ 




Ji 


E 


. D l 


J 2 




. ET 


J 2 



(17) 
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A matrix of the form 



where E = Di (Eq 12). This transformation 1 of J 2 is functionally equivalent to the original system 
(thus, its contraction limits remain) but it allows expression of the connection between the systems 
in the symmetric form of Eq 17. This functionally equivalent system can now be analyzed using 
the approach that follows. 

jj^r ? is negative definite if the individual systems Ji 2 are negative 
JJ J 2 

definite and if J 2 < D^J^D (Horn, 1985) (Page 472). Following (Slotine, 2003) (Section 3.4) and 
(Rutishauser et al., 2011) (Section 2.8), this implies that a sufficient condition for contraction is 
cr 2 (D) < A(Ji)A(J 2 ) where c(D) is the largest singular value of D and equivalent to ^4 in our case 
(all other elements of D are zero) and A is the contraction rate of the individual subsystems. Since 
the two subsystems are equivalent, the contraction rates are also the same Ai = A 2 = A(Ji ;2 ). It 
thus follows that the coupled systems are stable if $4 < X±. 

The contraction rate (Slotine, 2003; Rutishauser et al., 2011) of an individual subsystem is 
the absolute value of the largest eigenvalue of the Hermitian part of F (also see Eq (7)). It 
is Ai j2 = ||(— 2 + a)\ for our system. Thus the condition for the two coupled systems to be 
contracting reduces to 

ft < 1 - f (18) 
3.3.3 Summary of boundary conditions 

In summary, the following conditions guarantee stability of both the single and combined system 
as well as hard competition between the two coupled systems (that is, only one of the WTAs can 
have a winner). These conditions can be relaxed if a < 1, which will permit a soft winner-take-all. 



1< a < 2 y /p 1 p 2 3 
< /Wa < 1 

OL 

< ft < 1 - 2 (19) 

The lower bound on 04 is from the synchronization analysis, whereas the upper bound is from the 
stability analysis. These results illustrate the critical tradeoff between having enough strength to 
ensure functionality, while being weak enough to exclude instability. 

3.4 Speed of winner selection 

How quickly will a system select a winner? For a single WTA, this question is answered by 
how quickly a single system contracts toward a winner and for a coupled system how quickly the 
two systems synchronize. One of the key advantages of contraction analysis is that the rate of 
contraction, and in this case the rate of synchronization, can be calculated explicitly. We will 
express the contraction and synchronization rate in terms of the time constants r and its inverse, 
the decay constant, r refers to the mean lifetime of exponential decay x(t) = N^e' 1 ^ = Noe~ xt . 
A = - is the decay constant. Both the contraction and the synchronization rate are expressed 



1 The variable permutation is equivalent to a transformation of Ji by the permutation matrix 0: J 2 = 0Ji© 
" 1 " 
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in the form of a decay constant A. For example, the contraction rate of a system of the form 
N = —\N is equivalent to A. 

Physiologically, the time constants r in our system are experimentally determined membrane 
time constants that are typically in the range of 10-50ms (Koch, 1998; McCormick, Connors, 
Lighthall, & Prince, 1985; Koch, Rapp, & Segev, 1996; Brown, Fricke, & Perkel, 1981). For 
simplicity, we assume that all excitatory and inhibitory units have the same, but different, time 
constants te and 77, respectively. While the exact values depend on the cell type and state of the 
neurons, it is generally the case that 77 < te due to the smaller cell bodies of inhibitory neurons 
(McCormick et al., 1985; Koch, 1998). 

The bounds (19) were calculated assuming equal time constants for all units. However, the 
same calculations yield to very similar bounds when assuming different time constants for inhibitory 
and excitatory units (Appendix C, (Rutishauser et al., 2011)). In this case the ratio of the time 
constants ^ becomes an additional parameter for the parameter bounds. 

3.4.1 Speed of synchronization 

The synchronization rate is equivalent to the contraction rate of the system defined in Eq 13 (Pham 
& Slotine, 2007), which is the absolute value of the maximal eigenvalue of the Hermitian part of 
V'(RJc)V' T . Here, the original r" 1 is replaced by the diagonal matrix R, with the appropriate 
Te, Tj terms on the diagonal 2 . The matrix V is an orthonormal version of V as defined in Eq 14, 
which here is simply V = \/2~ 1 *V. 

The resulting synchronization rate (sync rate) is a function of the weights (local inhibitory 
loop) and $4 (remote inhibitory loop). We assume ^4 < ^3, which means that remote connectivity 
is weaker than local connectivity. However, qualitatively similar results can be found using the 
opposite assumption. For te = Tz", the sync rate is 

A s = ^(2-/3 3 + /3 4 ) (20) 

Note the tradeoff between local and remote connectivity: stronger remote connectivity increases 
and stronger local connectivity decreases the speed of synchronization (the larger A, the quicker 
the system synchronizes). For approximately equal connectivity strength (3 3 ~ /3 4 or in general for 
03, 04 <C 1, the sync rate is approximately r _1 . 

In general for t e 7^ 17, the sync rate is A s = TE+TI ^ Te 2r ^^ 1+ ^ d /?4 ' ) ^-. For example, 
for tj — ^ this reduces to A = ^7(3 — a/1 + 4(/3 3 — P4) 2 ). Again, for /3 3 ~ /3 4 , the sync rate is 
approximately tJ 1 . In conclusion, the sync rate is thus approximately equal to the contraction rate 
of the inhibitory units. Thus, synchronization occurs very quickly (20-50ms for typical membrane 
time constants). 

3.4.2 Speed of contraction 

The speed of selecting a winner (the contraction rate) for a single WTA can similarly be calculated 
based on the absolute value of the maximal eigenvalue of the Hermitian part of @(RJ)@ 1 (7). 
Assuming t = te = tj, the contraction rate is 

Ae = ^(2-a) (21) 



2 For the example of 3c (Eq 11), the diagonal terms are 77, 1 , r^ 1 , r 7 1 ,r B 1 ,r B 1 ,r J 1 
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Note that the larger a, the longer it takes till the system converges. Qualitatively similar findings 
result for other ratios of te and 77. For a typical value of a — 1.2 (see simulations below) and 
t = 20ms, the contraction rate would be 20s -1 . This equals a half-way time (time constant) of 
A -1 = 50ms. For a = 1.5, this would increase to 80ms. The time it takes to find a winner is thus 
a multiple of the membrane time constant (in this example 20ms) and substantially slower than 
the time it takes to synchronize the network. In conclusion, synchronization is achieved first which 
is then followed by winner selection. 

3.5 Coupling more than two WTAs 

So far we have shown how two different WTAs compete with each other after their inhibitory 
neurons are coupled. Similarly, more than two WTAs can compete with each other by all-to-all 
coupling of the inhibitory units, i.e. every WTA is connected with two (3^ connections from and 
to every other WTA. Thus, the wiring complexity of this system scales as 0(M 2 ) where M is the 
number of WTAs in the system (note that M is not the number of units but the number of WTAs). 
Notice also that the all-to-all coupling concerns only the sparse long-range excitatory connections 
and not the internal connectivity of the WTAs them-self. 

The same principle can be used to embed hierarchies or sequences of competition. Thus, in a 
network of such WTAs, some WTAs could be in direct competition with each other while others 
are not. Thus, for example, in a network of three WTAs A, B, and C relationships such as A 
competes with B and B competes with C are possible. In this case A does not directly compete 
with C. So if A has a winner, C can also have a winner. If B has a winner, however, neither B nor 
C can have a winner (see Fig 4D-F for a demonstration). 

Regardless of how many WTAs are combined and whether all compete with all or more selec- 
tively, the stability of the aggregated system is guaranteed if the individual sub-systems are stable 
and the coupling strengths /3 4 observe the derived bounds. While in themselves combinations of 
stable modules have no reason to be stable, certain combinations (such as the one we utilize) of 
contracting systems are guaranteed to be stable (Slotine & Lohmiller, 2001). This is a key benefit 
of Contraction Analysis for the analysis of neural circuits. 

3.6 Numerical simulations 

We simulated several cases of the network to illustrate its qualitative behavior. We used Euler 
integration with 5 = 0.01s. The analytically derived bounds offer a wide range of parameters for 
which stability as well as function is guaranteed. For the simulations, we chose parameters that 
verify all bounds discussed. 

First, we explored a simple system consisting of two WTAs with two possible winners each 
(Fig 3). Parameters were a = 1.2, fli = 2,/3 2 = 3, (3% = P4 = 0.1 and T = 0. We found that any 
of the four possible winners can compete with each other irrespective of whether they reside on 
the first or second WTA (Fig 3B-D shows an example). The inhibitory units quickly synchronized 
(Fig 3C) their activity and reached the same steady-state amplitude (because /3 3 = /3 4 ) 3 . 

Second, we simulated a system with 3 WTAs using the same parameters (Fig 4). For all-to-all 
coupling, all 3 WTAs directly compete with each other (Fig 4A,B), i.e. there can only be one 
winner across the entire system. Again, the inhibitory units all synchronize quickly during and 
after convergence (Fig 4C). We also simulated the same system with more selective connectivity, 

3 If /? 3 = /3 4 , it can be verified that x 3 (t) = 2/4 (t) for alH > if the initial values at t = are equal. Thus, the 
two inhibitory neurons become exactly equivalent in this special case. 
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Figure 3: Simulation of merged WTA consisting of two WTAs with two excitatory units each 
(four possible winners). (A) Illustration of connectivity and notation (color code). (B) Activity 
as a function of time, for two excitatory units on two different WTAs, together with the external 
input provided to the same units. Notice how the network selects the winner appropriately. (C) 
Activity of the inhibitory units in each WTA. Note: y% is slightly delayed for plotting purposes. 
(D) Activity of the two interconnect units. Notice how the output of the losing WTA descends 
to zero after the competition has been resolved and the network has contracted. Units of time are 
multiplies of the time constant r. Notice that the same color indicates the same unit throughout 
this figure (notation is shown in A). 
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eliminating competition between WTAs 1 and 3 (Fig 4D). This arrangement allows either one 
winner if it is on WTA 2, or two winners if they are on WTAs 1 and 3. If the maximal activity is 
not on WTA 2, then the network permits 2 winning states. Otherwise, if the maximal input is on 
WTA 2 only 1 winner is permitted (see Fig 4E for an illustration). This configuration allows for 
partial competition. 

4 Discussion 

Neural circuits commonly depend on negative feedback loops. Such recurrent inhibition is a crucial 
element of microcircuits from a wide range of species and brain structures (Shepherd & Grillner, 
2010) and enables populations of neurons to compute non-linear operations such as competition, 
decision making, gain control, filtering, and normalization. However, when considering biologically 
realistic versions of such circuits additional factors such as wiring length, specificity and complexity 
become pertinent. Here, we are principally concerned with the superficial layers of neocortex where 
the average distance of intracortical inhibitory connections is typically much shorter than the ex- 
citatory connections (Bock et al., 2011; Binzegger et al., 2004; Perin, Berger, & Markram, 2011; 
Katzel et al., 2011; Adesnik & Scanziani, 2010). In contrast, in invertebrates an inhibitory neuron 
has been identified that receives input from and projects back to all Kenyon cells (which are excita- 
tory) (Papadopoulou, Cassenaer, Nowotny, & Laurent, 2011). This neuron has been demonstrated 
to perform response normalization, making this system a direct experimental demonstration of 
competition through shared inhibition. No such system has yet been identified in the cortex. 

The number of excitatory neurons that can be contacted by an inhibitory neuron thus poses a 
limit on how many excitatory neurons can compete directly with one another (in terms of numbers 
and distance). Other models, such as those based on Mexican-hat type inhibitory surrounds (Hertz 
et al., 1991; Willshaw & Malsburg, 1976; Soltani & Koch, 2010), even require that inhibitory 
connectivity be longer range than the excitatory. These anatomical constraints have been used to 
argue that models such as the WTA are biologically unrealistic and as such of limited use. 

We have demonstrated here, by theoretical analysis and simulation, that it is possible to ex- 
tend such circuits by merging several independent circuits functionally, through synchronization of 
their inhibitory interneurons. This extension allows the construction of large, spatially distributed 
circuits that are composed of small pools of excitatory units that share an inhibitory neuron. We 
have applied and proved by non-linear contraction analysis that systems combined in this manner 
are inherently stable and that arbitrary aggregation by inhibitory synchrony of such sub-systems 
results in a stable system. This composition of subcircuits removes the limits on maximal circuit 
size imposed by anatomical wiring constraints on inhibitory connectivity, because the synchrony 
between local inhibitory neurons is achieved entirely by excitatory connectivity which can possi- 
bly be long-range so permitting competition between excitatory units that are separated by long 
distances; for example, in different cortical areas. We show that the time necessary to achieve 
sychronization is much shorter than the time required to select a winner. Thus, synchronization 
is faster than winner selection, which can thus proceed robustly across long-range connections 
that enforce synchronization. Further, selective synchronization between some WTAs but not oth- 
ers allows partial competition between some but not other WTAs (see Fig 4). The strength of 
these long-range connections could be modulated dynamically to enable/disable various competi- 
tions between two populations conditional on some other brain state. This modulation could be 
implemented by a state-dependent routing mechanism (Rutishauser & Douglas, 2009). 

There are several possibilities of mapping the abstract units in our model to real physiologi- 
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Figure 4: Simulation of merged WTA consisting of three WTAs with three possible winners each. 
(A-C) Case 1: Pairwise coupling allows all-to-all competition. (A) Illustration of connectivity. 
Filled circles are excitatory neurons, filled rectangles inhibitory. The activity of units with colored 
fills are shown as a function of time. (B) Activity of one excitatory unit for each WTA (bold 
lines) and the external input to each (dashed lines). Notice that there can be only one winner 
among the three WTAs. (C) Activity of the inhibitory units, shifted in time to each other slightly 
for plotting only. (D-F) Case 2: selective coupling, allowing partial competition only between 
1&2 and 2&3 but not 1&3. (E) Activity of the excitatory units for different cases. Notice that 
WTA 1 and 3 can have winners simultaneously, but not 2. Numbers indicate the WTA the winner 
belongs to. (F) Activity of the inhibitory units, illustrating synchrony in the presence of different 
absolute amplitudes. Units of time are multiplies of the time constant r. Notice that the same 
color indicates the same unit throughout this figure (notation is shown in A). 
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cal neurons. Our units are mean-rate approximations of a small group of neurons. In terms of 
intra-cortical inhibition, these would lie anatomically close to each other within superficial layers 
of neocortex. Since such inhibitory connectivity would have only limited reach, each inhibitory 
subunit can only enforce competition across a limited number of closeby excitatory units. Compe- 
tition between different areas is made possible by synchronizing remote populations by long-range 
excitatory mechanisms in the way we propose. Direct long-range inhibition, on the other hand, 
is unlikely both intracortically and subcortically, since all known connections from the thalamus 
and basal ganglia to cortex are excitatory. Networks such as the LEGION network (D. Wang & 
Terman, 1995) assume global inhibitory input to all excitatory units in the network, which for the 
reasons we discuss is unlikely in the case of cortex. It would, however, be possible to implement 
a feasible version of the global inhibitory input by synchronizing many local inhibitory neurons 
using the mechanism we describe, resulting in an anatomically realistic version of the LEGION 
network. 

Functionally, the model presented here makes several testable predictions. Consider a sensory 
area with clearly defined features as possible winners, such as orientations. The model predicts 
that the inhibitory units would not be tuned to these features, particularly if the number of 
possible winners is large. This is because the connectivity to the inhibitory units is not feature 
specific. Experimental studies indicate that this is indeed the case: units that functionally represent 
different tuning project to the same inhibitory unit, resulting in untuned inhibitory activity (Bock 
et al., 2011; Fino & Yuste, 2011; Kerlin, Andermann, Berezovskii, & Reid, 2010; Kuhlman, Tring, 
& Trachtenberg, 2011; Hofer et al., 2011). Secondly, this model predicts that inhibitory activity 
between two different areas or parts of the same area can either be highly synchronous or completely 
decoupled depending on whether at present the two are competing or functioning independently. 
This thus predicts that synchrony of inhibitory units should be affected by manipulations that 
manipulate competition, such as top-down attention. 

Our model suggests that synchronized populations of inhibitory neurons are crucial for enforcing 
competition across several subpopulations of excitatory neurons. It further suggests that the larger 
the number and spatial distribution of such synchronized inhibitory units, the larger the number 
of units that compete with each other. Experimentally, synchronized modulation of inhibitory 
neurons is a common phenomena that is believed to generate the prominent gamma rhythm trig- 
gered by sensory stimulation in many areas (Fries, Nikoli, & Singer, 2007; Whittington, Traub, & 
Jefferys, 1995; Traub, Whittington, Stanford, & Jefferys, 1996). Recent experiments have utilized 
stimulation of inhibitory neurons (Cardin et al., 2009; Sohal, Zhang, Yizhar, & Deisseroth, 2009; 
Szucs, Huerta, Rabinovich, & Selverston, 2009) to increase or decrease their synchronization with 
direct observable effects on nearby excitatory neurons such as, for example, increased or decreased 
amplitude and precision of evoked responses relative to how strongly the inhibitory neurons were 
synchronizing. Note that our proposal for this function of inhibitory synchrony is distinct and 
independent from the proposal that gamma-band synchrony serves to increase readout efficacy by 
making spikes arrive co-incidentally from a large number of distributed sources (Tiesinga, Fellous, 
& Sejnowski, 2008; Singer & Gray, 1995). Here, we propose that an additional function of such 
synchrony is to allow select populations of excitatory neurons to compete with each other because 
they each receive inhibition at the same time. 
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5 Appendix: Contraction Analysis 



This section provides a short summary of contraction analysis. We have previously published 
the detailed methods of applying contraction theory to WTA circuits (Rutishauser et al., 2011). 
Essentially a nonlinear time-varying dynamic system will be called contracting if arbitrary initial 
conditions or temporary disturbances are forgotten exponentially fast, i.e., if trajectories of the 
perturbed system return to their unperturbed behavior with an exponential convergence rate. A 
relatively simple algebraic conditions can be given for this stability-like property to be verified, 
and this property is preserved through basic system combinations and aggregations. 

A nonlinear contracting system has the following properties (Lohmiller & Slotine, 1998, 2000; 
Slotine, 2003; W. Wang & Slotine, 2005) 

• global exponential convergence and stability are guaranteed 

• convergence rates can be explicitly computed as eigenvalues of well-defined Hermitian ma- 
trices 

• combinations and aggregations of contracting systems are also contracting 

• robustness to variations in dynamics can be easily quantified 

Before stating the main contraction theorem, recall first the following properties: The symmet- 
ric part of a matrix A is A# = |(A + A* T ). A complex square matrix A is Hermitian if A T = A* 
, where T denotes matrix transposition and * complex conjugation. The Hermitian part Ah of any 
complex square matrix A is the Hermitian matrix |(A + A* T ) . All eigenvalues of a Hermitian 
matrix are real numbers. A Hermitian matrix A is said to be positive definite if all its eigenvalues 
are strictly positive. This condition implies in turn that for any non-zero real or complex vector 
x, x* T Ax > 0. A Hermitian matrix A is called negative definite if —A is positive definite. 

A Hermitian matrix A(x, t) dependent on state or time will be called uniformly positive definite 
if there exists a strictly positive constant such that for all states x and alH > the eigenvalues of 
A(x, t) remain larger than that constant. A similar definition holds for uniform negative definite- 
ness. 

Consider now a general dynamical system in M n , 

x = f(x,t) (22) 

with f a smooth non-linear function. The central result of Contraction Analysis, derived in (Lohmiller 
& Slotine, 1998) in both real and complex forms, can be stated as: 

Theorem Denote by ^ the Jacobian matrix of f with respect to x. Assume that there exists 
a complex square matrix ®(x,t) such that the Hermitian matrix &(x, t)* T @(x, t) is uniformly 
positive definite, and the Hermitian part F H of the matrix 

F =(6 + e£)e-> 

is uniformly negative definite. Then, all system trajectories converge exponentially to a single 
trajectory, with convergence rate | sup xt A max (F#)| > 0. The system is said to be contracting, F is 
called its generalized Jacobian, and &(x, t)* T @(x, t) its contraction metric. The contraction rate is 
the absolute value of the largest eigenvalue (closest to zero, although still negative) A = \X max F H \. 
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In the linear time-invariant case, a system is globally contracting if and only if it is strictly 
stable, and F can be chosen as a normal Jordan form of the system, with © a real matrix defining 
the coordinate transformation to that form (Lohmiller & Slotine, 1998). Alternatively, if the 
system is diagonalizable, F can be chosen as the diagonal form of the system, with © a complex 
matrix diagonalizing the system. In that case, F# is a diagonal matrix composed of the real parts 
of the eigenvalues of the original system matrix. 
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